fix: render standard-test assertion messages correctly by devin-ai-integration[bot] · Pull Request #1141 · airbytehq/airbyte-python-cdk

devin-ai-integration · 2026-09-01T15:08:34Z

Summary

Two assertion messages in the standard test suites never rendered their values, so test_spec / test_check failures printed literal braces instead of the counts that explain the failure.

Resolves (in part) https://github.com/airbytehq/airbyte-internal-issues/issues/17132:

https://github.com/airbytehq/airbyte-internal-issues/issues/17132

# connector_base.py — implicit concatenation where only the FIRST fragment had `f`
-    f"Expected exactly one CONNECTION_STATUS message. "
-    "Got: {result.connection_status_messages!s}"
+    "Expected exactly one CONNECTION_STATUS message. "
+    f"Got: {result.connection_status_messages!s}"

# source_base.py — no `f` prefix, and the message was a TUPLE
-    "Expected exactly 1 spec message but got {len(result.spec_messages)}",
-    result.errors,
+    f"Expected exactly 1 spec message but got {len(result.spec_messages)}. "
+    f"Errors: {result.errors!s}"

Behavior called out for review: in source_base.py the assert message was a tuple, which is always truthy as an assert message and printed as a tuple repr. It is now a single interpolated string that still includes result.errors. This goes slightly beyond the literal reported defect (the missing f), so flagging it explicitly rather than burying it — happy to split if you'd rather keep the tuple.

The new unit_tests/test/test_assertion_messages.py is a regression guard rather than a behavioral test: it AST-walks every module under airbyte_cdk/test/standard_tests and fails on any assert message that is a tuple, or that contains a {placeholder}-looking substring in a fragment missing the f prefix. Verified it fails on both pre-fix lines (connector_base.py:117, source_base.py:98) and passes after.

The other three defects in the issue are in airbytehq/airbyte (poe-tasks/*.toml) and are handled by a separate PR linked from the issue.

Test plan: poetry run pytest unit_tests/test -q → 124 passed; poetry run ruff check / ruff format --check clean on the touched paths.

Link to Devin session: https://app.devin.ai/sessions/7effc4c9e1fa40f1bf8f6c7aadac0d62
Open in Devin Desktop: https://app.devin.ai/desktop/session/7effc4c9e1fa40f1bf8f6c7aadac0d62?variant=devin

Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration · 2026-09-01T15:08:36Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-09-01T15:08:46Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1788275255-fix-standard-test-assert-messages#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1788275255-fix-standard-test-assert-messages

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/autofix - Fixes most formatting and linting issues
/poetry-lock - Updates poetry.lock file
/test - Runs connector tests with the updated CDK
/prerelease - Triggers a prerelease publish with default arguments
/poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
/poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

CDK API Reference

📝 Edit this welcome message.

Copilot

Pull request overview

This PR fixes two assertion messages in the CDK standard test suites that previously printed literal {...} instead of interpolated values, improving the usefulness of test_spec / test_check failure output. It also adds a unit-test guard to prevent similar assertion-message rendering mistakes from being introduced in airbyte_cdk/test/standard_tests.

Changes:

Fix connector_base.py assertion message by ensuring the fragment containing {result.connection_status_messages!s} is an f-string.
Fix source_base.py assertion message by replacing a tuple assert-message with a single interpolated string including result.errors.
Add an AST-based regression test to detect tuple assert messages and placeholder-like {...} substrings that would render literally.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
unit_tests/test/test_assertion_messages.py	Adds an AST-based regression guard against non-renderable assert messages in standard tests.
airbyte_cdk/test/standard_tests/source_base.py	Fixes `test_spec` assertion message to interpolate counts/errors correctly and avoids tuple messages.
airbyte_cdk/test/standard_tests/connector_base.py	Fixes `test_check` assertion message so the “Got:” portion interpolates correctly.

Suppressed comments (1)

unit_tests/test/test_assertion_messages.py:64

Same as above: reporting node.lineno will point at the assert statement rather than the start of the msg expression, making failures easier to locate.

            problems.append(
                f"{source_file.name}:{node.msg.lineno}: assert message contains "
                f"{placeholder} but the fragment is missing the `f` prefix"
            )

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-Authored-By: bot_apk <apk@cognition.ai>

github-actions · 2026-09-01T15:27:07Z

PyTest Results (Fast)

4 372 tests +9 4 360 ✅ +9 9m 30s ⏱️ -17s
1 suites ±0 12 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d21b65e. ± Comparison against base commit 64bdfc9.

github-actions · 2026-09-01T15:29:05Z

PyTest Results (Full)

4 375 tests 4 363 ✅ 11m 33s ⏱️
1 suites 12 💤
1 files 0 ❌

Results for commit d21b65e.

devin-ai-integration · 2026-09-01T15:55:21Z

CI note for reviewers: all required checks are green (Pytest 3.10–3.13, MyPy, Ruff lint/format, Build & Inspect). The one failure is the downstream Check: destination-motherduck connector test, which fails on check with a MotherDuck credential error rather than anything this diff touches:

AirbyteConnectionStatus(status=FAILED, message="An exception occurred: PermissionError(13, 'Permission denied')")
... Request failed: Your request is not authenticated. Please check your MotherDuck token.

The same job also failed on an unrelated recent CDK PR branch (devin/1787707125-movingwindow-honor-ratelimit-headers), so it looks environmental (secret/token availability) — but I can't fully rule it out from outside, so flagging it rather than dismissing it.

Written by Devin

fix: render standard-test assertion messages correctly

c31b999

Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration Bot added the hyd-fix label Sep 1, 2026

Airbyte Support Bot (airbyte-support-bot) requested a lite review from Copilot September 1, 2026 15:09

Copilot started reviewing on behalf of Airbyte Support Bot (airbyte-support-bot) September 1, 2026 15:10 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread unit_tests/test/test_assertion_messages.py

Comment thread unit_tests/test/test_assertion_messages.py

test: detect +-concatenated assert messages and report assert lineno

d21b65e

Co-Authored-By: bot_apk <apk@cognition.ai>

Navigation Menu

Uh oh!

fix: render standard-test assertion messages correctly - #1141

fix: render standard-test assertion messages correctly#1141
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1788275255-fix-standard-test-assert-messages

devin-ai-integration Bot commented Sep 1, 2026

Uh oh!

devin-ai-integration Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

devin-ai-integration Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

devin-ai-integration Bot commented Sep 1, 2026

Summary

Uh oh!

devin-ai-integration Bot commented Sep 1, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Sep 1, 2026

👋 Greetings, Airbyte Team Member!

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

PyTest Results (Fast)

Uh oh!

github-actions Bot commented Sep 1, 2026

PyTest Results (Full)

Uh oh!

devin-ai-integration Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant